Turning a Monolingual Speaker into Multilingual for a Mixed-language TTS

We propose a new approach to rendering speech of different languages with only a speaker’s monolingual recordings for mixed-code TTS applications. A reference speaker in the target language (say Chinese) is used to help building the target language TTS with “tiles” of the original source speaker‘s monolingual (say English) data. The difference between the monolingual source speaker and the reference speaker is firstly equalized by warping spectral frequency, adapting F0 dynamics and adjusting speaking rate accordingly. Thus equalized Chinese sentences of the reference speaker are tiled with the best tiles of the monolingual English speaker’s segments frame by frame. In additional to a standard English TTS which is trained with his monolingual English recordings, a Chinese TTS of the same speaker is trained with those newly tiled sentences. A mixedlanguage (English-Chinese) TTS is built to synthesize high quality, mixed-language (Chinese-English) speech in one consistent quality voice which is confirmed in both objective and subjective evaluations.